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ABSTRALT 



Assessment and Development of Student Probl em-Sol ving 
Skills Using Computer -man aged and Computer — assisted 
Instruction . 

Juchau , Wi 1 1 iam C. 1988: Major Applied Project, Nova 
University . Ed . D . Program in Computer Education . 

Descriptors : Computer Assisted Instruction /Computer 
Manager Ins true tion/Probl em-sol ving /Microcomputers/ 
Pascal Prog ramming /Ma t hematics/ Tutoring Systems . 

This pr ac ticum addressed the assessment and development 
of problem-sol ving skills. College faculty suggested 
students often lack a problem-solving strategy to apply 
to new probl ems . The current li ter ature provided no 
"best" way to teach problem-solving skills. Writers 
suggested "detached courses in problem-solving" and 

"PiRhPflflpfl pFRMPffi-*.frtv*nei srtrflteqw in wasting 

subj ec t area courses" as two possibl e solutions. 

This author devel oped a mathematics-oriented , 
computer-assisted instruction (CAI) program with an 
embedded general probl em-sol ving strategy . A rel a ted 
computer — managed instruction (CMI) module maintaineo 
student progress and "faded" the problem-solving "help" 
as students became skilled in solving the problems. 

Effectiveness was evaluated by pretesting and posttesting 
students who took the CAI program and control groups who 
received no CAI and by stud en t questionnai res , 

No statistically significant difference was found in 
the improvement of test scores between the experimental 
groups. The improvement in the mean test scores, 
however, of the group that that used the CAI program 
was about twice that of the control group. Results from 
the questionnaire showed that the students enjoyed using 
the CAI program and learned from it. Favorable 
improvement in the treatment group ' s test scores 
suggested that more r&s&arc h would be appropriate . 

The practicum'5 CAI and CMI program will be used by the 
author's college to find and help remediate shudents 
with poor problem-solving skills. 
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CHAPTER I 



INTRODUCTION 

Socio-Economic and Geographic Fac tors 

The author works as an Assistant Professor of 
Business Administration at a smal 1 , private col 1 ege . 
The coll ege offers three academic programs providing a 
liberal arts curriculum in accounting, arts, business 
administration, communications, drama, economics, music, 
nursing, religion, science, and several other fields. 
One program addresses the needs of traditional , 
four — year college students, and several other programs 
address the needs of part-time students, (Part-time 
students attending classes that meet one or two evenings 
each week must have an associate's degree and a full-time 
job.) The traditional college program enrolls about 
2,000 students. Nearly 700 associate's degree students 
enrol 1 in an evening , baccalaureate degree-completion 
program each semester. Additional ly, about 100 students 
enroll in the college's third program, an evening 
master's degree program in business administration. 
More mature students usual ly enrol 1 in the two evening 
programs. Their ages average about 30 years versus 

1 
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about 20 years in the traditional program. The college, 
located in a semi-urban city of about 100,000 
residents , rel ies heavi ly on the local community 
and state for students. Currently, only 30 percent of 
the students enroll from outside the state, and that 
percentage may decrease * The ci ty ' s major industries 
are agriculture, mining, small manufacturing, and 
tourism. Both the city and state are growing rapidly. 



Author's Work Setting and Role 



The author develops and teaches computer — related 
courses within this environment with primary 
responsibi 1 i ties in the computer information systems 
area , inc 1 uding in troduc tory computer courses , 
programming , systems analysis and design, database , 
project management, and related topics. The author 
teaches at the undergraduate and graduate levels in the 
day and evening programs , develops curriculum for 
computer — related courses, and often consults in academic 
and administrative computer systems devel opmen t and 
impl emen tat ion . 
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CHAPTER II 



STUDY OF THE PROBLEM 
Problem Description 

The college stresses teaching of problem-solving 
skills versus rote memorization of knowledge or facts . 
The Chairperson of the Psychology Department routinely 
determines students' problem-solving aptitudes with 
diagnostic tests. This testing has identified an 
important problem involving a portion of the 
students — particularly those in the evening program. 
Ideally, all students should have the same high level 
of probl em-sol ving skills. However , some of the evening 
program students display a lower skill level than 
the day program students. The department chair has 
verified this shortcoming by comparing test scores of 
the two groups over the last several years. The problem 
is to identify those students not possessing a high 
level of problem-solving skills and to correct the 
shortcoming . 
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Problem Documentation 



Diagnostic testing resul ts and interviews with 
faculty members provided evidence of the problem. As 
mentioned above, evening students appeared to test lower 
and to exhibit a lower degree of prob lem-so Iving ability 
in c lassroom assignmen ts . 

A faculty member, who teaches in the evening 

program , saw the prob 1 em as f o 1 lows : 

Candidates for the nursing degree, although 
highly motivated in their studies^ appear to 
lack some basic problem-solving skills. This is 
reflected, for example, in lower achievement on 
problem-solving skills tests when compared to 
college students in the other study programs. 
The reasons for this apparent shortfall cannot 
be ascertained . Perhaps , the candidates ' early 
undergraduate studies did not stress 
problem-solving activities or opportunities. The 
candidates ' work environment may primari ly be 
composed of conditions that stress lower level 
cognitive skills as opposed to problem-solving 
and decision-making opportunities . Sn any 
event, to ensure that the candidates have the 
appropriate skills when they leave the 
baccalaureate program, the college needs a means 
of identifying students with poor 

problem-solving ski 1 Is and a method of improving 
those skills where necessary . 

Causative Analysis and Related Literature 

This quote suggested sever a 1 possi b le causes of the 
problem of poor problem-solving skills. First, the 
students may 1 ack academic preparation in courses 
tending to improve problem-solving skills. Students in 
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the evening program tend to shun mathematics-oriented 
courses, or, for that matter, any course? traditionally 
requiring reasoning , logic , problem-solving , study , and 
hard work. They are, in this regard, similar to the day 
student. Piaget, a developmental psychologist, 
addressed the relationship of logical and mathematical 
structures to the process of cognition. The 
logico-mathematical structures form, according to 
Piaget, a system much like a "field of force" that 
clicks into operation when the student encounters a 
problem (Flavell, 1963). Secondly, the evening 
students' vocational work experience may seldom 
stimulate the aevelopment of good problem-solving 
skills. Many jobs encourage a "cookbook" or "check 
list" approach to problem-solving and discourage 
individual creativity in finding or developing new 
solutions. Nurses, for example, are not expected to 
seek innovative solutions to problems. They are often 
told to follow specified procedures an</ to seek help 
from a superior when the existing circumstances are not 
covered . Piaget ' s theories also apply here. Effective 
problem-solving requires thinking of all "possibilities" 
and "form" of solutions as wel 1 as the avai lable 
concrete facts. Dr. Leonard Huber (1985:43) reflected on 
Piaget ' s discussion of the "formal operational " thinking 
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patterns developed by adolescents and adults. 
"Before the invention of the printing press few 
people even reached the formal operational stage. 
They simply lacked the intellectual stimulation." 
Vocations void of problem-solving applications may 
foster similar results. Thirdly, the evening student 
may synthesize few of the problem-solving concepts from 
the various course offerings — a shortcoming resulting 
from the somewhat disjointed evening schedule and part- 
time staff. Course scheduling and staffing constraints 
combine with student work constraints to prevent the 
normal and preferred course progression that would 
maximize the student's synt esis of various course 
concepts. Thus, in the evening program, there is 
probablv less carryover of problem-solving concepts from 
one course to another (Stanger, 1982). Faculty employed 
full-time in the day program stabilize and integrate the 
curriculum by involving themselves in overall curriculum 
design an^ by teaching several integrated courses. This 
integrated environment may help traditional students 
synthesize problem-solving concepts . Conversely, the 
adjunct faculty members in the evening program lack the 
*,one degree of interest in the overall curriculum and, 
therefore, fail in the intt.r — disciplinary synthesis 
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necessary to transfer problem-solving ski 1 Is from one 
course to another. Students are left with the crucial 
job of synthesizing problem-solving skills introduced in 
various ways in many courses. Finally, even in an 
integrated environment, students may not transfer 
requisite problem-solving skills from one course or 
domain to another (Newel 1 , 1980) . For example, students 
may not transfer problem-solving skills that are learned 
in mathematics or chemistry c lasses to other courses 
(Gagne, 1980). Experts, repeatedly, use a "general 
problem-solving approach' 1 that can be applied to many 
different problems, while novices lack a universal 
"approach 11 to problem-solving (Chi, Feltovich, and 
Glaser, 1980). The college may err in thinking that 
the existing curriculum develops general problem-solving 
skills when it does not. 
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CHAPTER III 

EXPECTED OUTCOMES AND EVALUATION INSTRUMENTS 
S ta temen t of Genera 1 Goa 1 s 

The prac tic urn goa Is included identifying students 
lacking problem-solving skills and providing a method of 
improving those skills. The practicum outcomes included 
a method for identifying students who possess a low 
skill level in problem-solving and an effective 
educational computer program for improving those skills. 
Several instruments were used to evaluate the 
practicum' s outcomes. Two reasoning tests measured the 
success of the experimental CAI treatment in improving 
probl em-sol ving skills. Lowry-Lucier Reasoning Tests A 
and B served as pretest and posttest instruments, 
and a statistically significant improvement in the means 
was ex pec ted . The studen ts ' answers to a two-part 
written questionnaire given to the treatment group 
determined their recall and understanding of the 
embedded problem-solving approach. (For examples of 
these tests and questionnaires see Appendix A — Test 
and Evaluation Instruments. ) The questionnaire, 
given shortly after the CAI, contained a quiz that 
determined the students' knowledge of the generalized 

8 
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approach to problem-solving. Each student was expected 
to list, in the proper order and without error, the 
f our key words in the 4C ' s genera 1 prob 1 em-sol v ing 
strategy . Questions also addressed attitudes related 
to the practicum solution • The Statistical Package for 
the Social Sciences (SPSS), a statistics program, was 
used to analyze the pretest and posttest data. The 
nul 1 hypothesis suggested no statistical ly significant 
difference in the pretest and posttest means of the 
treatment group and the control group of students at a 
level of significance of 0.05, Analysis of variance 
and covar iance methods were used to ana lyze 
statistically the factors of student age, grade point 
average (GPA) , and SAT scores. 
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CHAPTER IV 

SOLUTION STRATEGY 

Discussion and Evaluation of Solutions 

This practicum integrated concepts learned in 
several study areas; however, the major concepts and 
solutions strategies stemmed from the study areas of 
Learn ing Theory , Courseware , and Programming . The 
study areas addressed concepts of teaching a general 
problem-solving strategy, suggested the use of CAI to 
assist in teaching the strategy, and provided the 
techniques and programming tools to create the CAI-CMI 
program* The Learning Theory area addressed teaching 
students a general purposs problem-solving strategy. 
Chi, Feltovich, and Glaser (1980) noted that "experts 11 
approach problems with a particular problem-solving 
style, or heuristic, that they evidently learn and 
novices lack. Derry and Murphy (1986) stated that 
research impl ied the existence of domain -genera 1 ( versus 
domain -specific ) heuristic planning models , or 
"metastrategies , 11 that are effective in problem- 
solving . For example , a general model of 
problem-solving might entail steps in (a) analysis and 
goal identification 9 ( b) planning a strategy , (c ) 

10 
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strategy , and ( e ) chang ing the strategy. Baron ( 1981 ) 
stated that a major advantage of training sue h a 
problem-solving model was that the strategy would help 
students to control their "stylistic 11 propensities in 
thinking. The impulsive student, according to Baron, 
could be trained to benefit from more reflective 
thinking in appropriate circumstances, Metacognitive 
experiences, or thinking about how we learn, can improve 
the acquisition and use of learned skills (Flavell, 
1981), Although more research is needed, the conclusion 
that training in metastrategy techniques has produced 
modest, statistically significant gains in student 
academic performance is empirically supported (Dansereau 
et al., 1979). Unfortunately, no agreement existed cn 
the "best 11 model for problem-solving, and numerous 
problem-solving models were available. Derry and 
Murphy ( 1986 ) reported that more than twelve different 
metastrategy models are popularized in various forms of 
training. Kellis (1985) analyzed many existing 
problem-solving models and found four conceptually 
similar "steps" present in ten different models. The 
prol iferation of these numerous problem-solving models 
or metastrategies could generate instructional 
conditions that are confusing and interfering. This 
carrying out the strategy, (d) checking results of the 
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practicum implemented the Four C's Learning Plan (4C's) 
model suggested by Derry and Murphy (1986). The four C's 
are as follows: (1) Clarify the problem (identify goals, 
alternatives, and constraints), (2) Choose an 
alternative (costs versus benefits analysis of 
alternatives, and decision-making), (3) Ce^rry out the 
dec is ion , ( 4 ) Check the resul ts ( evaluation , review , and 
iteration). Models similar to the above problem-solving 
strategy have been suggested for diverse areas such as 
educational curriculum design, computer systems 
analysis, and military mission planning. While not a 
panacea, the above 4C's model presented a useful 
problem-solving model with wide application. 
Several possibilities existed for training in a 
general problem-solving strategy. One approach treated 
the strategy as a "stand-alone" curriculum. Here, 
"teaching a problem-solving strategy" became the course 
objective replacing the teaching of specific subject matter. 
A danger of this "detached" training method is that the 
strategies themselves are addressed at the lowest level 
of learning skills. The problem-solving strategies are 
reduced to lists to be memorized — level one on Bloom's 
learning hierarchy (Bloom, J956). Embedding the 
problem-solving strategy within the academic subject or 
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basic ski 1 Is area provided a better solution. This 
approach linked the problem-solving strategies to the 
rea 1 -world , subject matter environment ( Sternberg , 
1983 ) . Thus , mathematics teachers woul d use the 
metastrategies to explain specific problem-solving 
techniques, while language arts teachers would explain 
how reading and memorization tac tics fit into the 
strategies, and computer science teachers would explain 
computer systems deve 1 opmen t using the same 
problem-solving framework (Derry and Murphy, 1986). 
However, embedded training in problem-solving may result 
in a loss of "generality" since the student must 
transfer the problem-solving strategies to other subject 
areas. The preferred approach is likely a compromise 
combination of the detached and embedded approac hes to 
training in problem-solving strategies (Beyer, 1987) • 
Thus , one a 1 ternative so lution to the prac ticum 
problem was to create a formal course in prob 1 em-sol ving 
strategies. The general problem-solving strategies 
could be introduced into the curriculum in this course 
and reinforced within the subject area courses. This 
solution would entail adding another course to an 
already ful 1 schedule and making other curriculum 
revisions. The Courseware study area suggested how the 




14 

computer might effectively help solve the practicum 
problem. Another option existed for developing a 
computer-assisted instruction ( CAI ) program with a 
problem-solving strategy (the 4C's) embedded within the 
sub jec t-matter course or ex ere ise • The strength of this 
approach wou Id be the extended use of problem-solving 
skills in a realistic subject-natter context (Jones et 
al . , 1985) • This approach would require no change to 
the curriculum since the CAI program could be used 
individually outside the formal course offerings, 
"urthermore , the integration of a computed-managed 
instruction (CMI) module with the CAI program would 
automatically maintain student progress and provide data 
for identifying students wi ch problem-solving or 
subject-matter deficiencies. Derry and Murphy (1986) 
described a job-skills CAI program (JSEP) using a system 
controlled prompting method that (1) prompted students 
at selected points to recall and use certain 
problem-solving techniques; (2) analyzed student 
performance to help determine when prompts were no 
longer needed; and (3) gradually phased out prompts in 
advanced stages of in s true t ion when evidence ex is ted 
that the student was spontaneously applying 
problem-solving strategies . 
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Description of Selec ted Solution 

Based on the time and cost constraints associated 
with the curriculum, a CAI approach similar to the above 
JSEP program appeared most appropriate as the prac ticum 
solution. The Psychology Department used a manual, 
mathematics-oriented problem-solving test to assist in 
assessing student skills. Table 1 shows examples of 



Table 1 



Examples of XYZ Proolems 



1. Z 2. X 3. X 4. X 
+ X Y X Y 
+ Y + X + Y 



XY 



YY YX XX 



the problems which required the student to substitute 
a singular, integer value from zero through nine for the 
letters X, Y, or Z . The problems were constructed such 
that only one combination of values resulted in a correct 
solution. Although the problems would appear to favor 
algebraic solutions, the fastest way to solve the 
problems entails the development of a series of heuristic 
techniques (based on such concepts as arithmetic carry, 
answer approximation, and minimum or maximum values) 
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that apply to the problems. The existing test format, 
familiar to the department faculty, formed the basis for 
a CAI -CM I program entitled Computer — assisted 
Problem-solving (CAPS) . 

/ 

Report of Action Taken 

A prototype screen format and user interface was 
created during the CAI program design. This "throwaway" 
prototype tested the users' (i.e., representatives of the 
Psychology Department) acceptance of the screen layouts. 



! CAPS: A Problem Solving Exercise [Diagnostic Problems]! 



=== ! PRC/3LEM !==== ===== ! SOLUTION !=== ===== ; CHOICES ! =- 



Z 

+x 

XY 



4 
+ 1 

15 



X = ? 
Y = 5 
Z = 4 



===! Problem 1 !== ===== ! Problem 1 ! == =============^== 

======================= { DIRECTIONS ! =================== 

Please enter the correct values ! 
Press the <Esc> key for each value to skip the problem 
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Figure 1 

Example of Screen Layout for CAI Program 
(Modified to fit on this page and print without graphics.) 
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Figure 1 shows the program ' s screen format. Ful 1 -scale 
program design efforts started after acceptance of the 
screen layouts and user interface . This* prototype 
approach used good system and program design techn iques 
by ensuring that the output was in the desired format 
before making extensive coding efforts. Each program 
task was then identified and recorded in a hierarchy 
chart (or program structure chart). 



+ + 

! CAPS Main ! 
! Program ! 
+ ! + 



+ + + 

! Opening ! 
! Logos ! 
+ + 



+===+====+ 
! Maintain ! 
! Progress ! 



+ + + 

! Teachers ! 
! Utility ! 
+ + 



+ + + 

SClosing I 
! Chores ! 
+ + 



Student! 
Logon ! 
+ 



+ ! + 

lExample ! 
IProblem ! 
+ + 



! Diagnostic ! 
! Problems ! 
+ + 



! Problems ! 
! Presentation ! 



IShow the 
! Tutorial 
! on Demand 
!of Student 
+ 



-+ + ! + + j + + j + 



!Pop Up the 
IHint, Skill 
!& Strategy 
! Windows 
+ 



STime & Check 
! Student for 
! Appropriate 
j Feedback 
+- 



Update the 
Student ' s 
History 
File 



+ + + + + + + 



Figure 2 



CAPS Program Hierarchy Chart (Structure Chart) 
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Figure 2 shows the final hierarchy chart for the main 
program. 



WHILE Student is not finished 

! WHILE First Time for Exercises 

! ! IF (Diagnostic NOT taken) AND not Escape DO 

! ! ! Diagnostic Test 

! ! END IF 

! ! IF a diagnostic probl em was missed 

! ! ! Answer first diagnostic probl em missed 

! ! ENDIF 

! ENDWHILE (First time for exercise) 

! If'MlLE Problem is missed AND not escape 

! ! Answer first problem not correct 

! ! Go to next probl em not answered correc t ly 

! ENDWHILE 

! WHILE Skills bad AND not escape DO 

! ! Answer ,'andom problems related to specific skills 

! ENDWHILE 

! WHILE Bugs present AND not escape DO 

! ! Answer random problems related to specific bugs 

! ENDWHILE 

! WHILE ( a problem not in standards ) AND not escape 

! ! Answer substandard problems again 

! ENDWHILE 

ENDWHILE (not escape) 
Update Files 
EXIT to DOS 



Figure 3 

Example Program Pseudocode (Progress Module) 
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Pseudocode and design documentation was created for each 
program task. An example of the pseudocode is shown in 
Figure 3. The program was written in the Pascal 
language using modular, top-down programming techniques 
to enhance program reliability and main tainabi 1 i ty , 

The embedded problem-solving strategy was presented 
as a tutorial on the 4C's method of problem-solving and 
as M pop-up windows" that appear periodically during the 
exercises . This brief 4C ' s problem-solving tutorial was 
recallable on demand (by pressing "t") throughout the 
program. The pop-up windows appear as a function of the 
student's responses to the exercises. The student's 
time-on-task , number-of -tries , and choice-of -answer were 
used to determine the proper help and strategy guidance 
to provide the student via the windows. The CM1 portion 
of the program enabled automatic student tracking and 
skill level recording. " In tel 1 igent' 1 CAI (ICAI) 
concepts assisted in creating the CMI module (Rambally, 
1986). Specifically, the program used a listing cf 
skills and ''logic bugs" (i.e., erroneous approaches to 
solving the problems) that were fundamental to the 
mastery of the subject area material . Here, the 
skills and bugs dealt primarily with finding heuristic 
techniques for solving the problems. 
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SKILLS; 

Status Skill Description 



3 Considering MAXIMUM values of numbers. 

-3 Considering MINIMUM values of numbers. 

3 Using correct CARRY value to next column. 

0 Choosing ODD OR EVEN values where needed. 

3 Using APPROXIMATION of the answer. 

-1 Using SUBGOALS to help solve the problem. 

LOGIC BUGS: 

Status Bug Description 



-1 Entering numbers that DO NOT ADD UP. 

-3 Using SAME NUMBER for different letters. 

1 MULTIPLYING not adding for solution. 



Figure 4 

Skills and Logic Bugs Used in CAPS CAI Program 
(Condensed from Teacher's Utility Module) 



Figure 4 shows the author's l?st as implemented in the 
CMI module. As the student responded to the problems, 
the appropriate skill or bug "status" was adjusted by 
plus or minus 1 accordingly. A status range of -3 to +3 
was used in the CAI program to provide remedial problem 
presentation and was reported to the teacher through 
the CMI module. A teacher module was integrated into 
the CMI module of the program. Holding down the "Alt" 
key on the computer keyboard and pressing "222" cn the 
numeric keypad activates a menu of options that a teacher 
can use to moni tor and manage students . 
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Student DIAGNOSTIC TEST Statistics 

Last Name: < JUCHAU > Password: < 1 > 

Date Started: 4-16-88 Data File: STUDENT. DTA 

Problem Number ==> 12 3 4 

Number of Tries: 3 13 1 

□n Task Time [sec]: 29 1 3 1 

Problem Right?[=l]: 0 0 0 0 

Correct Ans: 0 Total Task Time: 34.0 

7* Correct: 0.0 Total Window Time: 0.0 

Avg Time of Correct Ans: 0.0 Total Exercise Tinra: 34.0 

Press <Esc> to EXIT Press the <SPACE BAR> to CONTINUE 

Figure 5 

Teacher ' s Report Showing Student Diagnostic Resul ts 
( Compressed from Teacher ' s Uti 1 i ty Module ) 
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JUCHAU > 




Pass 


word 




< 1 
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: 4- 


16-88 




Data 


File: 


STUDENT. 


DTA 


Problem ==> 


1 


2 


3 
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5 




6 


7 


8 


Tries : 


5 


5 


0 


0 




0 




0 


0 


0 


TaskCsec ] : 


30 


19 


0 


0 




0 
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0 


0 


Window : 


75 


83 


0 
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0 
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RightOl] : 
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14 
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0 


0 




0 




0 
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0 


0 


0 




0 




0 








0 


0 


0 


0 




0 




0 








0 


0 


0 


0 




0 




0 






Correct Ans: 


0 






Total 


Task 






0.82 


mins. 


'/. Correct: 


0.0 






Total 


Window 


« 


2.63 


mins . 


Avg Time of 


Correct Ans 


: 0.0 


Total 


Exerc ise : 


3.45 


mins . 


Press <Esc> 


to EXIT 


Press 


the 


<SPACE 


BAR> 


to CONTINUE 



Figure 6 

Teacher's Report Showing Student Exercise Results 
(Extracted from Teacher's Utility Module) 



ERIC 



27 



22 



Figures 5 and 6 show the first two teacher's screens. 

Selected faculty and students tested the program 
during development. These test personne 1 provided 
comments and suggestions for improved or modified 
versions of the program. The testing also provided 
standards of difficulty for the various CAI problems 
used in later program development. In October of 
1987, the CAI program was field tested with a group of 
twelve students. Two shortcomings were identi f ied 
during the test. First, the students were not reading 
the "help" windows that were presented to them. In 
their eagerness to "get on with the problems , 11 they 
eye led quick 1 y through the pop-up windows that provided 
hints and the general problem-solving strategy. Second, 
there was no way for the students to return to the 4C's 
tutorial once it had been viewed. Both of these 
problems were addressed in revisions to the program. 

During March and April 1988, an experiment was 
conducted using groups of students randomly selected 
by the college registrar. The treatment group was 
composed of two intact sections of about 44 students 
enrolled in an introductory computer course. The 
control group consisted of about the same number 
of students enrolled in a psychology course. Both 
groups took the pretest and post test . The treatment 
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group had a two-week period before the post test during 
which they were told to use the CAPS program (to assist 
in developing a general problem-solving strategy ) . The 
treatment group completed a questionnaire after the 
post test. The resul ts and analysis of the experiment 
and questionnaire are presen ted in Chapter V. 
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CHAPTER V 

RESULTS, IMPLICATIONS, AND RECOMMENDATIONS 

Evaluating the practicum objectives of identifying 
and removing shortf al Is in student problem-solving skil Is 
required attention to several important and different 
educational outcomes. First, the CAI-CMI program was to 
provide a means to d iagnose and remed iate student 
problem-solving deficiencies. To meet this goal the 
CAI-CMI program had to be easy to use, error free, and 
pedagogical ly sound. The effectiveness of this 
outcome was measured by student opinions expressed in a 
questionnaire and through examination of the program's 
content and operation. Secondly, the program attempted 
to 11 teach, 11 in a somewhat subliminal mode, a general 
problem-solving approach. A brief "quiz," administered 
with the questionnaire, verified accomplishment of this 
outcome. Finally, the program was to improve 
problem-sol ving skills. ^n experiment tested the 
effectiveness of the CAI program as well as the general 
problem-sol ving strategy . 
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Results 



The practicum implementation created a CAI-CMI 
program , entitled Computer — assisted Problem Solving 
(CAPS). The program's design and coding consumed 
two man-years of effort. The CAI component of the 
program used modern CAI screen design techniques from 
writers such as Jesse M. Heines (Heines: 1984). All CAI 
screens, for example, were divided into functional areas 
(e.g., orientation information, directions, responses, 
error messages, etc.) providing the user a consistent 
interface throughout the program. "Pop-up windows' 1 
presented information without cluttering the screen and 
without breaking the program's continuity. This window 
technique presented the general problem-solving 
strategy and reinforced it during the solving of the 
mathematics subject area exercises. Concurrently, the 
CMI program component provided ease and flexibility in 
student tracking and conserved Cw..*puter memory by 
utilizing a linked-list, dynamic data structure for the 
student data base. The executable CAI-CMI program 
(CAPS), that contains over 5000 lines of source code, 
is provided on a 5 1/4" diskette as an enc losure to this 
report • The program runs on an I BM or compatib le 
computer with MGA or CGA video adapters, 256KB memory, 
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and one 360KB disk drive. Details for obtaining the 
source code are provided in Appendix C. 

Statistical analysis of the experiment used the 
microcomputer version of the Statistical Package for 
Social Sciences (SPSS/PC+ ) . 



Table 2 

Summary Statistics for Pretest and Posttest Scores 



PRETEST POSTTEST 





Control 


Treatment 


Control 


Treatmen 


Cases 


37 


23 


37 


23 


Mean 


16.270 


13.087 


18.378 


17.087 


Std Err 


.550 


.793 


.665 


.873 


Median 


16 


13 


18 


17 


Mode 


14 


9 


15 


15 


Std Dev 


3.347 


3.801 


4.044 


4.188 


Variance 


11.203 


14.447 


16.353 


17.538 


Kurtosis 


-.030 


.337 


-.846 


-1.071 


S E Kurt 


.759 


.935 


.759 


.935 


Skewness 


-.16- 


-.198 


-.201 


-.075 


S E Skew 


.388 


.481 


.388 


.481 


Range 


15 


17 


15 


14 


Minimum 


9 


4 


9 


10 


Maximum 


24 


21 


24 


24 



Summary statistics for the pretest ana posttest scores 
are shown in Table 2. 
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PLOT OF PRETEST WITH POSTTEST BY GROUP 



P 
R 
E 
T 
E 
S 
T 



16+ 



8+ 



0+ 



2 

1 1 



1 2 

2 2 

2 2 $ 1 
2 

2 2 



1 



2 $ 2 $ 2 
2 2 
1 2 

1 



2 2 2 2 
2 $ 
2 2 
112 
1 



- + + H + - 

11.25 15.75 
9 13.5 



+ 16 



+8 



+0 

— + + + +- 

20.25 24.75 
18 22.5 



POSTTEST 



Notes: 60 cases plotted. 

1 = Plotting symbol of Treatment Group. 

2 = Plotting symbol of Control Group. 
$ = Multiple occurrence of data. 



Figure 7 

Scatterplot of Pretest and Posttest Scores 
by Treatment and Control Group 



Figure 7 is a scatterplot of the test scores by 
treatment and control groups. 
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Table 3 



Statistical T Test Results 
Independent Samples of Groups 
Separate Variance Estimate 





PRETEST 


POSTTEST 


DIFFERENCE 


t Value 


-3.30 


-1.18 


1.60 


Degrees of Freedom 


42.31 


45.54 


42.88 


2-Tail Prob. 


.002 


.245 


.118 



Table 3 presents the results of statistical analysis 
using the Student's t test. 



Table 4 

Correlation Coef f ic ien ts for Selected Variables 



Corr 


: PRE 


POST 


DIFF 


AGE 


GPA 


SAT 


PRE 


1. 


.3570* - 


.5589** 


.0298 


.3069 


.3431* 


POST 


.3570* 


1. 


.5750** 


.1270 


.2619 


.4582** 


DIFF 


-.5589* 


.5750* 1 




.0867 


-.0364 


.1062 


AGE 


.0298 


.1270 


.0867 


1. 


. 1731 


.0878 


GPA 


.3069 


.2619 


.0364 


.1731 


1. 


.3181* 


SAT 


.3431* 


.4582* 


. 1062 


.0878 


.3181 


1. 


N of 


cases: 55 


1-tailed 


Signif : 


* - . 


01 ** 


- .001 



Table 4 provides the correlation coefficients for other 
variables that may affect the experiment's outcome. 
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Table 5 

Student Questionnaire Responses 





Strong ly Agree Don ' t Know Disagree 


Strongly 




Agree 


Disagree 


1 - 


I am comfortable when using a computer. 






177. 617. 167. 67. 


07. 


2. 


Mathematics is one of my stronger subjects 






177. 507. 07. 177. 


177. 


3. 


The instructions for solving the problems 


were clear 




287. 567. 177. 07. 


07. 


4. 


The program had more than one objective. 






117. 397. 447. 67. 


07. 


5. 


I improved at solving the problems as I progressed. 




07. 617. 227. 11% 


67. 


6. 


The problems were too hard. 






07. 67. 397. 507. 


67. 


7. 


The problems were too easy . 






67. 117. 397. 337. 


117. 


8. 


The "windows" or "hints" were distrac ting . 






07. 287. 177. 397. 


177. 


9. 


The "hints" were helpful . 






117. 507. 117. 117. 


177. 


10. 


I read the "hints." 






227. 567. 67. 117. 


67. 


11 . 


I was challenged by the problems. 






227. 617. 117. 07. 


67. 


12. 


I had fun doing the problems. 






177. 567. 117. 67. 


117. 


13. 


I understood the objective of the program. 






67. 337. 50% 07. 


11% 


14. 


The "hints" were always the same . 






07. 11% 397. 447. 


67. 


15. 


I felt I was being tested. 






6% 287. 177. 447. 


67. 


16. 


The mechanics of entering the answers were 


simple. 




287. 447. 117. 11% 


67. 


17. 


I could easily see the results of my choices. 




337. 507. 117. 07. 


6/. 


18. 


The math content was much too easy. 






67. 177. 287. 287. 


227. 


19. 


I enjoyed doing the problems. 






177. 507. 117. 117. 


J 17. 


20. 


I learned from the exerc ise . 






07. 447. 397. 07. 


177. 
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Table 5 is a summary of student responses to the 
questionnaire (Appendix A) given to the treatment group. 
A copy of the Subject Consent and Release Form used in 
the experiment is in Appendix B. 

Table 6 

General Strategy Quiz Resul ts 



Provided Correct Percent Responding with # 

Terms Response 12 3 4 



Connecting 






127. 






Cataloging 




67. 


127. 




127. 


Choosing 


2 




*177. 






Carrying 


3 






*67. 




Classifying 




127. 


447. 


67. 




CI arif ying 


1 


*337. 


127. 


67. 




Correcting 








127. 


447. 


Checking 


4 






447. 


*447. 



Capitulating 



Note: Total of 59 percent of the students selected the 
correct terms, but not all in the correct order. 

The results of a quiz (Appendix A) given to students in 
the experimental treatment group are shown in Table 6. 
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Impl ications 



Robert M. Gagne (1977:36) , addressing the subject 

of learning problem-solving skills, stated that 

Surely, not enough is yet known about how 
to arrange the conditions of learning so that 
effective cognitive strategies will be learned. 
For one thing, the strategies themselves have 
not been identified and described — most of 
them are obviously more complex than our example 
of the mnemonic system. It seem probable, too, 
that a learner ' s cognitive strategies improve in 
increments over long periods of time, rather 
than being totally learned in a few days, weeks 
or months. 

In order to solve a problem, the learner must be 
able to reca 1 1 relevant rules that have been 
previously learned, (p. 162) 

The evidence of experimental studies concerning 
the use of discovery in problem solving 
certainly does not demonstrate that higher-order 
rules must be learned by discovery (see Ausubel , 
1968, pp. 471-473). In much adult learning, for 
example, the guidance provided by verbal 
instructions is so complete that the rule tc be 
learned is stated verbally during the course of 
learning . ( p. 164) 

.... 

To summarize , discovery or problem solving 
involves the combining of previously learned 
rules into a new higher — order rule, which 
"solves" the problem and generalizes to . . . 
other problems of the same type. Problem 
sol ving occurs when the i nst rue t ions provided 
the learner do not include a verbally stated 
"solution," but require the construction of such 
a solution "on one's own." . . . Bat the 
capability acquired by this means does not 
appear to differ in a fundamental sense from 
that acquired when instructions include the 
statement of a "solution" (unless, of course, 
the latter is learned simply as a verbal chain). 
What is learned in either case is a higher — order 
rule, which is based upon some previously 
learned simpler rules . (p. 165 ) 
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CAPS , as implemented in the prac ticum , attempted to 
address both the identification of the higher-order 
rules of problem -sol ving and the discovery of their 
application through a problem-solving exercise. 

The CAPS program was meant to be "user — friendly 1 ' 
— a design concept not easily implemented . 
The results of the student questionnaire seemed to show 
achievement of the design goa 1 „ For example , 72 percent 
of the students found it simple to enter the problems' 
answers (Question 16), and 84 percent found the 
instructions clear (Question 3). Most students could 
see easily the result of their choices (Question 17 — 
83 percent). Three quarters of the students had fun 
doing the problems (Question 12 — 73 percent), and only 
a third felt they were being tested (Question 15 — 34 
percent) . 

Of equal concern , was the goal of effectively 
presenting the general problem-solving strategy without 
distracting the student from the subject area material. 
Clearly, the windows thai exist for presentation of 
general problem-solving strategies and other hel p must 
not overshadow the subject area objectives of the CAI 
program. Only 28 percent found the "hint" windows 
distracting or not helpful (Questions 8 and 9). 
Conversely, 61 percent found the windows helpful. About 
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half the students recognized the program's inherent dual 
objectives (Questions 4 and 13). The students did not 
expect a quiz on the M 4C" words and, therefore, did not 
attempt to memorize them. As such, the quiz results (in 
Table 6 above ) are commendable . Nearly 60 percent of 
the students could recall the unordered words. However, 
knowledge of the words — or even of the general 
problem-solving concepts — are low-level educational 
objectives (Bloom, 1956). The desired educational 
outcome was the student's ability to use the 
problem-solving method in new, diverse environments. 
The practicum experiment attempted to test the program's 
effectiveness in meeting this objec tive . 

The experiment, as described in Chapter III and 
summarized above, attempted to demonstrate that the 
treatment group (or those taking the CA T program) would 
do better on later problem-solving t;isks. 
Statistical analysis of the experiment's data does not 
clearly support this assumption. Unfortunately, a 
significant difference exists in the pretest means of 
the control and treatment groups. This should not be 
if both groups were randomly sampled from an 
homogeneous population. The flaw in experimental design 
resulted most likely from th«? groups being pre-existent 
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or intact. No significant difference occurred in the 
posttest means of the groups. An ideal experiment 
should begin with equal group means and, after 
treatment, end with a significant difference in the 
mean scores of the groups . In this respect , the 
experiment appears to have failed — at least it failed 
to reject the null hypothesis that there was no 
statistical ly significant difference in the groups 
after the treatment and allow acceptance of the 
alternative hypothesis. However, the experimental data 
d.\d support, at the 88 percent confidence level, a 
significant "improvement" in the test score means. The 
treatment group started with a lower mean and improved 
significantly more than the control group (See Tables 2 
and 3). The gravity of a Type I error may not warrant 
accepting an 88 percent confidence level, but it 
suggt 'ts careful review of the data to preclude blind 
acceptance of the statistical f indinqs and rejection of 
the null hypothesis. There was a significant 
correlation of grade point average ( GPA ) , and 
Scholastic Aptitude Test (SAT) scores with pretest and 
posttest scores (See Table 4). These variables were 
intuitively linked to problem-solving skill level, the 
experimental data demonstrated a statistically 
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significance correlation. As expected, there was a 
statistical ly significant correlation between the 
pretest and posttest scores. 



The CAPS program is a working CAI-CMI program with 
potential for addressing the practicum objectives. CAPS 
used current and effective CAI techniques to present a 
general problem-solving strategy embedded within a 
subject area. The CMI module allowed the tracking of 
student progress and skill level for diagnostic and 
remed ial purposes . More , however , needs to be done to 
refine the CAPS program and to show that it effectively 
and efficiently solves the practicum* s stated problem. 
First, experimental study of the CAPS program (or its 
revisions) should continue. A larger, randomly selected 
group of students should be used for the next control 
and treatment groups. Samples paired by age, sex, GPA, 
SAT, and major would help to control some of the 
intervening variables associated with the experimental 
design. The CAPS program should be revised to make the 
subject area component intrinsically more motivating. 
The algebraic XYZ problems were interesting, but 
astaining student interest and, therefore, attention 
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might be better achieved by matching the subject area 
to students' particular study areas or interests, 

A logical extension of the CAPS program is a 
CAPS-like authoring system that would allow an 
educator to select the subject area problems, required 
skills, and logic bugs associated with a particular 
environmen t . The environmen t wou Id not have to be 
mathematics-or ien ted * Many educational areas, such as 
business or economics , could benefit from the 
application of a general problem-solving strategy, A 
"generative" CAI module (Hoffman, 1975) that develops 
new subject area problems of various difficulties based 
on student skills and predetermined algorithms for 
problem creation (and problem-solving) would be an even 
more valuable addition to the CAPS program. 



The author's college will benefit from the 
practicum in several ways. Since no method existed for 
identifying students with deficiencies x problem-solving 
skills, CAPS will provide the Social Science Division a 
means of diagnosing and tracking these students. Not 
enough is known of methods for improving problem-solving 
skills, and educators may be spending too little time 
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addressing the problem. Dissemination of the CAPS 
program within the college will no doubt cause 
significant debate as to the merits of a general 
problem-solving strategy and the best means for teaching 
i 4 : to students. Finally, if faculty can be inspired to 
modify the subject area content and use the CAPS program 
design in their courses the practicum will be 
perpetuated . 
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APPENDIX A 
EXAMPLE TEST AND EVALUATION INSTRUMENTS 
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(Extract from) 
LOWRY-LUCIER REASONING 
Test A 



DIRECTIONS: Each answer is a day of the week- Sunday 
is always the first day of the week for answers to 
problems . For Sunday , write the number 1 in the answer 
column- For Tuesday, write the number 3 in the answer 
column. The number corresponding to each day of the 
week is listed on the top of each sheet. 



1. Sun 2. Mon 3. Tues 4. Wed 5. Thur 6. Fri 7. Fri 

A. If today were Saturday, what would tomorrow be?. .( > 

B. If today were the first day of the week, what was 
yesterday? , ( ) 

C. If yesterday were Saturday, what day is tomorrow? ( ) 

D. If yesterday were Sunday, what was the day before 
yesterday? ( ) 



There are 25 questions that get progressively harder. 
For example, number 24 reads: 

24. If the days were reversed, and if today were 
Saturday, and if the day before yesterday were the sixth 
day of the week, then what would be the day before the 
first day of the week? ( ) 
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(Extract from ) 
LOWRY-LUCIER REASONING 
Test B 

DIRECTIONS: The answer is always a number, or two or 
three numbers. Examples A, B, and C will be worked with 
you. Ask any questions about the problems at that time. 
No questions wi 1 1 be answered whi le the test is being 
taken . 




Imagine the little squares 
at the left made with 
matches . How many mate hes 
must be removed so that 
the square numbered " 1" 
will be gone, but the 
other two will remain 
complete? (....) 



B . How many mate hes must be 
removed so that square 
"2 11 will be gone leaving 
the other two compl ete? 
( ) 



C. By removing two matches, only, which square will be 
entirely gone, leaving two complete squares and nothing 
else? ( ) 



Again, the 25 questions get progressively harder. Number 
25 is shown below: 





1 


2 






3 


4 


5 


6 




7 


8 


9 


10 


11 



25. What is the smallest sum of three squares that can 
be eliminated by removing three matches? (....) 
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Questionnaire 



Thank you for participating in the experimental project. 
Your ef f or ts and time are sincerely appreciated . Please 
answer the following questions by inserting a number 
from 1 to 5 that expresses your honest opinion based on 
the tol lowing scale: Do not put your name on the paper! 



1 . I am comfortable when using a computer . 

2. Mathematics is one of my stronger subjects. 

3. The instructions for solving the problems were 
clear. 

4. The program had more than one objective. 

5. I improved at solving the problems as I 
progressed . 

6. The problems were too hard. 

7. The problems were too easy. 

8. The "windows" or "hints" were distracting. 

9. The "hints" were helpful. 

10. I read the "hints." 

11. I was challenged by the problems. 

12. I had fun doir g the problems. 

13. I understood the objective of the program. 

14. The "hints" were always the same. 

15. I felt I was being tested. 

16. The mechanics of entering the answers were simple. 

17 . I could easi ly see the resul ts of my choices . 

18. The mathematics content was too easy. 

19. I enjoyed doing the problems. 

20. I learned from the exercise. 

Please put a number in f ront of the "4C words" to 
show their order of execution in problem solving. 

Pick only four. 

Connecting 

Cataloging 

Choosing 

Carrying 

Classifying 

Clarifying 

Correcting 

Checking 

Capitulating 



5 

Strong ly 
Agree 



4 . . 

Agree 



. 3 

Don ' t Know 



Disagree Strong ly 
Disagree 



2 1 
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SUBJECT CONSENT AND RELEASE FORM 
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TO: The Registrar 



I grant Mr. Juchau permission to use my age, GPA, and 
SAT (or equivalent score) data for the purpose of an 
educational research project. I understand that the 
data wi 1 1 be kept confidential and once the group 
statistics have been created will be destroyed. My name 
or other personal data wi 1 1 not be used in the research 
findings on an individual basis. 



Signed 
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DETAILS FOR OBTAINING C.A.P.S. PROGRAM 
SOURCE CODE LISTINGS 
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Source code is not included in this report, 
but it can be obtained from the author for a 
nominal hand 1 ing fee . Address inquiries to : 

Wi 1 1 iam C. Juchau 
2303 Cambridge Avenue 
Lakeland, FL, 33803 

Ple^ase inc lude a sel f -addressed , stamped 
envelope. 
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